A portion of the disclosure of this patent document contains material to which a claim of copyright protection is made. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but reserves all other rights whatsoever.
1. Field of the Invention
The present invention relates to graphical programming, and in particular to a system and method for automatically generating a graphical program to perform an image processing algorithm.
2. Description of the Related Art
Traditionally, high level text-based programming languages have been used by programmers in writing applications programs. Many different high level programming languages exist, including BASIC, C, FORTRAN, Pascal, COBOL, ADA, APL, etc. Programs written in these high level languages are translated to the machine language level by translators known as compilers or interpreters. The high level programming languages in this level, as well as the assembly language level, are referred to as text-based programming environments.
Increasingly computers are required to be used and programmed by those who are not highly trained in computer programming techniques. When traditional text-based programming environments are used, the user""s programming skills and ability to interact with the computer system often become a limiting factor in the achievement of optimal utilization of the computer system.
There are numerous subtle complexities which a user must master before he can efficiently program a computer system in a text-based environment. The task of programming a computer system to model or implement a process often is further complicated by the fact that a sequence of mathematical formulas, mathematical steps or other procedures customarily used to conceptually model a process often does not closely correspond to the traditional text-based programming techniques used to program a computer system to model such a process. In other words, the requirement that a user program in a text-based programming environment places a level of abstraction between the user""s conceptualization of the solution and the implementation of a method that accomplishes this solution in a computer program. Thus, a user often must substantially master different skills in order to both conceptually model a system and then to program a computer to model that system. Since a user often is not fully proficient in techniques for programming a computer system in a text-based environment to implement his model, the efficiency with which the computer system can be utilized to perform such modeling often is reduced.
Examples of fields in which computer systems are employed to model and/or control physical systems are the fields of instrumentation, process control, industrial automation, and simulation. Computer modeling or control of devices such as instruments or industrial automation hardware has become increasingly desirable in view of the increasing complexity and variety of instruments and devices available for use. However, due to the wide variety of possible testing/control situations and environments, and also the wide array of instruments or devices available, it is often necessary for a user to develop a program to control a desired system. As discussed above, computer programs used to control such systems had to be written in conventional text-based programming languages such as, for example, assembly language, C, FORTRAN, BASIC, or Pascal. Traditional users of these systems, however, often were not highly trained in programming techniques and, in addition, traditional text-based programming languages were not sufficiently intuitive to allow users to use these languages without training. Therefore, implementation of such systems frequently required the involvement of a programmer to write software for control and analysis of instrumentation or industrial automation data. Thus, development and maintenance of the software elements in these systems often proved to be difficult.
U.S. Pat. Nos. 4,901,221; 4,914,568; 5,291,587; 5,301,301; and 5,301,336; among others, to Kodosky et al disclose a graphical system and method for modeling a process, i.e., a graphical programming environment which enables a user to easily and intuitively model a process. The graphical programming environment disclosed in kodosky et al can be considered the highest and most intuitive way in which to interact with a computer. A graphically based programming environment can be represented at a level above text-based high level programming languages such as C, Pascal, etc. The method disclosed in Kodosky et al allows a user to construct a diagram using a block diagram editor, such that the diagram created graphically displays a procedure or method for accomplishing a certain result, such as manipulating one or more input variables to produce one or more output variables. In response to the user constructing a data flow diagram or graphical program using the block diagram editor, data structures are automatically constructed which characterize an execution procedure which corresponds to the displayed procedure. The graphical program may be compiled or interpreted by a computer. Therefore, a user can create a computer program solely by using a graphically based programming environment. This graphically based programming environment may be used for creating virtual instrumentation systems, industrial automation systems, modeling processes, and simulation, as well as for any type of general programming.
Therefore, Kodosky et al teaches a graphical programming environment wherein a user places or manipulates icons in a block diagram using a block diagram editor to create a graphical xe2x80x9cprogram.xe2x80x9d A graphical program for controlling or modeling devices, such as instruments, processes or industrial automation hardware, may be referred to as a virtual instrument (VI). In creating a virtual instrument, a user may create a front panel or user interface panel. The front panel includes various user interface elements or front panel objects, such as controls or indicators, that represent or display the respective input and output that will be used by the graphical program or VI, and may include other icons which represent devices being controlled. The front panel may be comprised in a single window of user interface elements, or may comprise a plurality of individual windows each having a user interface element, wherein the individual windows may optionally be tiled together. When the controls and indicators are created in the front panel, corresponding icons or terminals may be automatically created in the block diagram by the block diagram editor. Alternatively, the user can place terminal icons in the block diagram which may cause the display of corresponding front panel objects in the front panel, either at edit time or later at run time. As another example, the front panel objects, e.g., the GUI, may be embedded in the block diagram.
During creation of the graphical program, the user may select various function nodes or icons that accomplish his desired result and connect the function nodes together. For example, the function nodes may be connected in a data flow or control flow format. The function nodes may be connected between the terminals of the respective controls and indicators. Thus the user may create or assemble a data flow program, referred to as a block diagram, representing the graphical data flow which accomplishes his desired process. The assembled graphical program may then be compiled or interpreted to produce machine language that accomplishes the desired method or process as shown in the block diagram.
A user may input data to a virtual instrument using front panel controls. This input data may propagate through the data flow block diagram or graphical program and appear as changes on the output indicators. In an instrumentation application, the front panel can be analogized to the front panel of an instrument. In an industrial automation application the front panel can be analogized to the MMI (Man Machine Interface) of a device. The user may adjust the controls on the front panel to affect the input and view the output on the respective indicators. Alternatively, the front panel may be used merely to view the input and output, and the input may not be interactively manipulable by the user during program execution.
Thus, graphical programming has become a powerful tool available to programmers. Graphical programming environments such as the National Instruments LabVIEW product have become very popular. Tools such as LabVIEW have greatly increased the productivity of programmers, and increasing numbers of programmers are using graphical programming environments to develop their software applications. In particular, graphical programming tools are being used for test and measurement, data acquisition, process control, man machine interface (MMI), supervisory control and data acquisition (SCADA) applications, simulation, and machine vision applications, among others.
Although graphical programming environments may reduce the complexity and time involved in performing general programming tasks, it is sometimes desirable to provide prototyping environments specialized for creating certain types of programs or algorithms even more easily. One example of such a specialized prototyping environment relates to the field of image processing. The term xe2x80x9cimage processingxe2x80x9d is used herein to refer to image processing, image analysis, and machine vision, among others. Image processing prototyping environments have been developed for rapidly developing image applications, e.g., for machine vision, pattern matching, shape matching, or other applications. Such an image processing prototyping environment may enable a user to easily and quickly experiment with an image, applying various image processing functions in order to manipulate, process, or analyze the image.
However, in many cases users need to perform additional operations not supported by an image processing prototyping environment or need to incorporate an image processing algorithm into a larger program. In such cases, it may be desirable to enable the image processing prototyping environment to automatically generate a program to perform the image processing algorithm developed by the user. In particular, for the reasons highlighted above, it may be desirable to generate a graphical program that performs the image processing algorithm, which the user may then modify using a graphical programming environment.
The problems outlined above may in large part be solved by providing a system and method for automatically generating a program to perform an image processing algorithm, as described herein. A user may develop an image processing algorithm (or prototype) in an image prototyping environment. The image prototyping environment may provide image processing functions for analyzing, processing, or manipulating various types of images, such as binary, grayscale, color, or complex images. The image prototyping environment may include a user interface enabling the user to easily apply the image processing functions to an image and immediately see the results, in order to develop the desired algorithm. In various embodiments, any of various types of image processing functions may be supported, including filtering functions, morphology functions, histogram functions, particle analysis functions, edge detection functions, etc. As the user applies each image processing function to an image, the function may be recorded as a step in a script or recipe (also called a prototype). A script essentially specifies an algorithm; i.e., an algorithm may be defined by the plurality of steps in a script.
Once the user has developed a script, the user may be able to test the algorithm specified by the script on other images, and make any desired changes to the script. When the user is satisfied with the algorithm, the user may request the image prototyping environment to automatically generate a program implementing the image processing algorithm. In various embodiments, the prototyping environment may be operable to generate different types of programs, including text-based and graphical programs. For example, a text-based program generated in response to a script recorded while developing the image processing algorithm may include function calls corresponding to each step in the script, while a graphical program generated in response to the script may include various nodes corresponding to the steps of the script. The generated program may then be executed to perform the algorithm.
In one embodiment, a code generation xe2x80x9cwizardxe2x80x9d is operable to display a series of interrelated user interface screens or windows, in response to the user requesting the program to be generated from the script. The wizard may be operable to receive user input specifying various options to be used in generating the program. For example, the user may select a particular programming language to use in generating the program and/or a particular programming development environment where the program will be used. The user may also select an image source or type of image source with which the program will operate, e.g., an image file or an image acquired from a hardware device. In one embodiment, the image processing functions of the script may have associated parameters, and the user may specify one or more of these parameters that are desired to be interactively changeable or viewable. The generated program is then operable to display a user interface including controls that enable the user to change or view the parameter values interactively.
In one embodiment, an association between the script created in the prototyping environment and the program generated from the script may be maintained. This association may enable a user to run the automatically generated program and then return to the prototyping environment in order to view or edit the script used to generate the program. In one embodiment, the generated program may be xe2x80x9clockedxe2x80x9d, requiring the user to explicitly unlock the program before it can be modified. Locking the program may facilitate the retrieval or recreation of the script information used to generate the program.
As noted above, in various embodiments, a graphical program may be automatically generated from a script developed in the prototyping environment. One embodiment of a method for automatically generating such a graphical program is described. The prototyping environment may interface with a graphical programming environment in order to generate the graphical program, wherein the graphical programming environment provides an application programming interface (API) to support the dynamic creation/editing of a graphical program. One particular embodiment of such an API is described.